Combining virtual reality and tactile stimulation to investigate embodied finger-based numerical representations

Finger-based representation of numbers is a high-level cognitive strategy to assist numerical and arithmetic processing in children and adults. It is unclear whether this paradigm builds on simple perceptual features or comprises several attributes through embodiment. Here we describe the development and initial testing of an experimental setup to study embodiment during a finger-based numerical task using Virtual Reality (VR) and a low-cost tactile stimulator that is easy to build. Using VR allows us to create new ways to study finger-based numerical representation using a virtual hand that can be manipulated in ways our hand cannot, such as decoupling tactile and visual stimuli. The goal is to present a new methodology that can allow researchers to study embodiment through this new approach, maybe shedding new light on the cognitive strategy behind the finger-based representation of numbers. In this case, a critical methodological requirement is delivering precisely targeted sensory stimuli to specific effectors while simultaneously recording their behavior and engaging the participant in a simulated experience. We tested the device’s capability by stimulating users in different experimental configurations. Results indicate that our device delivers reliable tactile stimulation to all fingers of a participant’s hand without losing motion tracking quality during an ongoing task. This is reflected by an accuracy of over 95% in participants detecting stimulation of a single finger or multiple fingers in sequential stimulation as indicated by experiments with sixteen participants. We discuss possible application scenarios, explain how to apply our methodology to study the embodiment of finger-based numerical representations and other high-level cognitive functions, and discuss potential further developments of the device based on the data obtained in our testing.

• Configure the names in the project. In the menu Edit -> Project Settings -> Player chose a Company Name, a Product Name and a Version (can use the one already there).
• In the same window, in the Other Settings tab, find the Minimum API Level and select Android 6.0 Marshmallow (API level 23).
Leave the Target API Level as Automatic (highest installed).
• Still in the Other Settings tab, under Configuration, change the Scripting Backend to IL2CPP. Clear the ARMv7 checkbox and select ARM64.
• Go to Edit -> Project Settings -> XR Plugin Management and click Install XR Plugin Management.
• After installing, in the Android tab, select Oculus.
• Click the PC tab (red circle on the image) and select Oculus.
• Go back to Edit -> Project Settings -> Player and then the Other Settings tab. Under the Rendering option, change the Color Space to Linear.
• Clear the Auto Graphics API checkbox and make sure OpenGLES3 is the first one in the Graphics API below.
• Make sure Multithreaded Rendering is selected.
• Go back to Edit -> Project Settings -> XR Plugin Management and click the submenu Oculus, on the left.
• Select the version of the Oculus Quest that will be used and check the Low Overhead Mode (GLES). Disable all other checkboxes.
• The last step is to configure the Quality of the project. Access Edit -> Project Settings -> Quality and click on Medium (should have a green checkmark under the Android icon).
• Set Pixel Light Count to 1. • Import the downloaded package using the menu Assets -> Import Package -> Custom Package.
• Select the downloaded unitypackage file.
• Import all assets.
• In the Project tab, navigate to the Hand Experiment folder, then Scenes and open the BaseRoomScene.

Importing the Unity Project Files
• In the Hierarchy tab, on the left side of the screen, expand the OVRCameraRig by clicking on the arrow next to it. Expand TrackingSpace, then expand LeftHandAnchor. Select OVRCustomHandPrefab_L. In the Inspector tab, on the right side of the screen, click "Auto Map Bones". The list below should fill with finger names.
• Do the same to the OVRCustomHandPrefab_R, under RightHandAnchor.
• The application is now ready to be tested inside Unity.
• Press Play on the Unity Editor. The Game tab should show the blackboard seen from the floor (this happens because Unity is not detecting a VR Headset, so it positions the camera on the ground).
• It is possible to click Maximize on Play in the Game tab before pressing play to have a full screen view.
• There is some text on the top left corner of the screen to help debug the connection with the Experiment Controller in the next steps. This will not show in the VR headset.
• The application has some shortcuts to enable testing on the Unity Editor. Start the application for a test run by clicking inside the game window (represented in the image by the red rectangle) and pressing S.
• After seeing the message about the dominant hand, press R to simulate a right-handed individual or L to simulate a left-handed individual.
• Next, the blackboard will display a plus sign. This signals the participant to rest their hand on the ball or table.
• Press A to simulate that the participant has rested their hand. This will deliver the tactile stimulation and show a minus sign on the screen, showing the application is waiting for the participant's gesture.
• Press Z to simulate that the gesture was made by the participant. This will trigger the visual stimulation part of the experiment.
• Press C to simulate that the participant has given their verbal response so the application moves to the next trial, going back to the +.
• After going trough all the trials (8, in the example), the experiment will be finished, displaying a "thank you" message.
• Note that the last three trials have no visual cue, as they are supposed to use the participant virtual hand as the visual cue, which depends on the Oculus Quest being connected to work.
• In those trials, the numbers used to generate the fake virtual hand and the motor stimuli (explained later in this Supplementary Material) are instead shown on the blackboard (this will be disabled after testing).
• After reaching the last screen, press Play again to stop the application.
• The next step is to test the network communication between the Experiment Controller and the VR application.
• Open two instances of Unity, one with the Experiment Controller and other with VR appliation. Connect the Arduino to the PC as described in Supplementary Material 1. Press Play on both instances.
• Be very mindful of which instance is active when you press the keyboard shortcuts. The keypress will only be identified by the Unity that is active when it happens.
• With both projects on Play mode, select the Experiment Controller and press S. This should send a signal to the VR application telling it to start.
• If everything worked correctly, a Start message will be displayed on the top left corner of the VR application (as shown on the image to the right) and the application will swap to the Dominant Hand part.
• After once again selecting the Experiment Controller, press R for the right hand or L for the left hand. This will display another message in the VR application and will start it.
• To test the connection back, go to the VR Application (it should be displaying a + sign) and press A. This should trigger the first tactile stimulation request, vibrating the motors on the Arduino and lighting the LED, validating the other side of the communication (from the VR application to the Controller).
• To test all the trials, the A, Z, C, combination mentioned before can be used on the VR application (with its window selected) so it keeps sending stimulation requests to the Experiment Controller.

Troubleshooting Network Communication
• If no messages were displayed in the VR application, first make sure both are running and that the Experiment Controller was selected when pressing S to start.
• In the Hierarchy, click on the Network Controller in both applications. Check to see if both have 127.0.0.1 as the IP and the ports are matching correctly -the Send in one application must be the Receive in the other, as described in Supplementary Material 1, and vice-versa.
• If still no messages are seen, try to use different ports than those selected (5300 and 5301, for example). Remember to stop both applications before changing those values and then pressing Play again.
• If the messages are still blocked, try to contact the network admin about blocked UDP ports.

Preparing the VR Headset
• If this is the first use of the Oculus Quest VR headset, it needs to be configured using a mobile app. Download the Oculus Mobile App on an Android or iOS phone and follow the in-app instructions to sign.
• Pair the headset with the app following the instructions on screen and conclude the setup process through the Oculus Quest headset. Make sure to also update it to the latest version.
• The official website has a guide for setting up and updating the device: https://support.oculus.com/articles/getting-started/getting-started-with-quest-2/index-gettingstarted-quest-2 Preparing the VR Headset • With a USB-C cable, connect the device to the PC (it is also possible to use a USB-C to USB cable). A popup should show on the device asking to allow data. Click Allow, in the device.
• In Unity, go to File -> Build Settings (Ctrl+Shift+B on Windows). On Run Device, the headset should now be showing as an option. Select it.

Preparing the Application for the VR Headset
• There are two different ways the application can be run on the VR headset. The easiest way is to use Oculus Link to simulate a tethered VR headset. This way, the application can be run from Unity directly to the headset, as if it was being run on the PC. This makes network communication more trivial and the application execution and testing easier. This will be described in the next steps.
• The second way is to compile the application as a .apk file and install it in the headset as a native application. This will require a network connection between the VR headset and the PC, the correct configuration of both devices IPs in the Network Controller component in both and a redeploy of the application file every time a change is made in the application code. The advantage is that the headset can be totally wireless, and the application can be started without the need for a PC, after installing, directly on the VR headset.
• To run using this method, first configure both Network Controllers with correct IPs and ports as mentioned in the Supplementary Material 1, then select the Run Device as mentioned before and click Build and Run. Select the APK name and Unity should install it in the device.

Using the Oculus Link to Run the Application in Unity
• For the Oculus to work directly inside Unity, the Oculus Link PC app is needed. Unplug the headset then download and install the software on the PC from the official website: https://www.oculus.com/setup/ • After installing the software, plug the headset back to the PC and you may be prompted with three popups: • DENY data access, this time.
• ALLOW USB debugging from the PC.
• ENABLE the Oculus Link.
• You may unplug the headset and plug it again if a different option is selected.
• The official page has some extra troubleshooting if needed: https://support.oculus.com/articles/headsets-andaccessories/oculus-link/connect-link-with-quest-2/ • In the VR headset, Go to Quick Settings and select Oculus Link. The VR headset should now be ready to run directly from the Unity Editor.
• When ready to test, connect the Tactile Device to the PC and the motors to the user's fingers, run the Experiment Controller and press play. Afterwards, run another Unity instance with the VR Application and press Play. The VR Application should now be displaying in the VR headset.
• Raise both hands in front of the headset to detect them. Afterwards, select the Experiment Controller and press S to start the application and then press R or L to select the dominant hand.

Using the Oculus Link to Run the Application in Unity
• Rest (curl) the selected dominant hand to generate a tactile stimulation.
• Make the gesture according to the tactile stimuli to trigger the visual stimuli.
• Say the number on the screen to activate the voice threshold and enable the answer input.
• The application will wait for the input of the answer. Select the Experiment Controller on the PC and press 1-6 on the keyboard to record the number as the answer -6 should represent any not possible answer.
• Rest the hand again for the next stimuli. This is the cycle for a single trial.
• The sample file contains 8 trials.
• Now that the application is working, the next step is to customize it.

Configuring the Voice Detector Threshold
• If the application is to be used in a noisy environment or on a quiet one, the voice detection can vary. If needed, it is possible to change the detectable threshold by going to the Voice Detector component in the Hierarchy and changing the value for the threshold. Test the application again to find one that is good enough.
• To do so, press play with the Voice Detector component selected. It will display the values of noise it is detecting in the Test Sound field.
Change the threshold to one that only triggers when something is said close to the mic. Change this after STOPPING the application, as changes in the play mode are automatically reverted afterwards.
• All the trials are directly imported from a trial file in the Assets -> Resources folder. Customizing this trials file is the easiest way to change the application.
• The trial file, shown in the image, is a .csv file containing 5 elements for each trial -the trial ID, the motor stimulus, the format of the visual stimulus, the visual stimulus to display and the canonicity of the visual stimulus. The software uses this file to determine which image to show on the black board and how to deliver the tactile stimulation.

Configuring the Trials List
• Trial ID is a number identifier for each trial.
• Motor Stimuli tells the Tactile Stimulation Device which fingers to vibrate (1 being on and 0 being off, from thumb to pinky).
• The Visual Stimulation Format must be one of the following: • Picture -Shows an image of the hand on the blackboard. • 3D Model -Shows a 3D model of the hand on the blackboard. • Arabic Digit -Shows a single digit on the blackboard.
• VR Hand -Enables the finger swapping and shows the virtual copy of the user's hand.
• Visual Stimulation is the number to show on the blackboard, using the format in the previous column. It can be 2, 3 or 4 for Picture, 3D Model, and Arabic Digit formats and must be a hand configuration (described later) for the VR Hand condition.
• Canonical or Non-Canonical describes if the pose to be displayed, when relevant, is the canonical or non-canonical one. This column accepts C, NC or NA (when not applicable).
• For example, if the format is Picture, the visual stimulation is 2 and the canonical is C, the image on the right will be displayed.
• The VR Hand stimulation format enables the finger swapping in the virtual hand, as described in the paper. This swap occurs according to the values in the Visual Stimulation field.
• The field takes 5 numbers, representing the virtual fingers from thumb to pinky. The first number will describe which real finger controls the virtual thumb, the second number which real finger controls the index, and so on. Each number ranges from 0 to 5, with 1 being the thumb and 5 the pinky and 0 being used to make the virtual finger immobile. DO NOT use spaces between the numbers.